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Abstract. Let V be a complex linear space, G C GL(V) be a compact group. 
We consider the problem of description of polynomial hulls Gv for orbits Gv, 
v £ V, assuming that the identity component of G is a torus T. The paper 
contains a universal construction for orbits which satisfy the inclusion Gv C 
T c v and a characterization of pairs (G, V) such that it is true for a generic 
v £ V. The hull of a finite union of T-orbits in T c v can be distinguished in 
closT c i> by a finite collection of inequalities of the type |2i| si . . . |2n| s " < c. 
In particular, this is true for Gv. If powers in the monomials are independent 
of v, Gv C T c v for a generic v, and either the center of G is finite or T c has an 
open orbit, then the space V and the group G are products of standard ones; 
the latter means that G = S n T, where S n is the group of all permutations 
of coordinates and T is either T™ or SU(n) n T™, where T™ is the torus of all 
diagonal matrices in U(")- The paper also contains a description of polynomial 
hulls for orbits of isotropy groups of bounded symmetric domains. This result 
is already known, but we formulate it in a different form and supply with a 
shorter proof. 



Introduction 

Let V be a finite-dimensional complex linear space and G C GL(V) be a compact 
subgroup of GTj(V). We consider the problem of description of polynomially convex 
hulls for orbits O v — Gv, v £V . The polynomially convex hull (or polynomial hull) 
Q of a compact set Q c V is defined as 

(0.1) Q = {zeV : |p(«)|<sup|p(C)| for all p£V{V)}, 

where V(V) is the algebra of all holomorphic polynomials on V . It is usually 
difficult to find Q. For Q = Gv, the answer is known if G is an isotropy group of a 
bounded symmetric domain in C n . Paper [9j contains a description of G-invariant 
polynomially convex compact sets, including hulls of orbits (Q C V is polynomially 
convex if Q = Q); it continues paper [TD] and uses results of [S]. On the other 
hand, it is known that an orbit of a compact linear group is polynomially convex 
if and only if the complex orbit G c v is closed and Gv is its real form ((7J)- The 
cases G — U(2),SU(2) were considered in pQ, [4]. The problem of determination 
of polynomial hulls of orbits admits the following natural generalization: given a 
homogeneous space M of a compact group G, describe maximal ideal spaces M.a of 
G-invariant closed subalgebras A of C(M), where G(M) is the Banach algebra of all 
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continuous complex-valued functions on M with the sup-norm. If A is generated by 
a finite-dimensional invariant subspace, then Ma can be realized as the polynomial 
hull of an orbit. Paper [B] contains a description of Ma for bi-invariant algebras 
on compact groups and partial results on spherical homogeneous spaces. Maximal 
ideal spaces for U(«)-invariant algebras on spheres in C™ are described in [IT]. 

In this paper we consider orbits Gv of groups G = FT, where F C G is a finite 
subgroup and T is a torus, such that G c v = T c v. Let t C gl(V) be the Lie algebra 
of T and set t R = it, T R = exp(t R ). Suppose that v £ V has a trivial stable 
subgroup in T and let X C T R v be finite. The hull of Y = TX admits a simple 
description. If X = {v}, then Y = Tv is the closure of Texp(Cr)f, where Ct is 
a cone m t R . If T c is closed, then Y = Texp(Q x )v, where Q x Q t R is a convex 
polytope (the convex hull of the inverse image of X for the mapping £ — > exp(£)v, 
£ £ t R ). Any segment in Qx corresponds to an analytic strip or an annulus in 
Y. In general, Y is the union of Tu, where u runs over exp(Qx)^- Also, Y is 
distinguished in clos T c v by a finite family of monomial inequalities of the type 

(0.2) \ Zl \ s K..\z n \ s » <c, 

where c > and s = (si, . . . , s n ) £ ffi" depend on u and X. Vectors s correspond 
to normals of faces of Ct + Qx- 

Thus, the problem of determination of Gv is not difficult if Gv C T c v. The 
latter is equivalent to the assumption that the complex orbit G c v is connected. In 
Example 13.41 we give a construction for orbits which satisfy this condition; here is 
a sketch. The group G — FT acts on the space V = C(K), where if is a finite 
F-invariant subset of t*: F acts naturally on C(K), t = t** is naturally embedded 
into C(K), and T = exp(t) acts on C(K) by multiplication. If v £ G(K) is an 
F-invariant function, then Gv C T c v. According to Theorem 13.51 each connected 
complex orbit can be realized in this way. Further, we describe pairs (V, G) such 
that 

(0.3) Gv C T c v for a generic v £ V. 

By Theorem I4.3) under the additional assumption that the complex linear span of 
T c v coincides with V, this happens if and only if the group G C Z, where Z is the 
centralizer of G in GL(V), has an open orbit in V. There are two extreme cases: 
(A) Z C G c ; (B) G has a finite center. An example for (A) is the group G = S n T n 
acting in C n , where T™ is the torus of all diagonal matrices in ~U(n) and S n is the 
group of all permutations of coordinates. Replacing T" with SU(ro) flT™, we get an 
example for (B). Example 14.41 contains a construction for pairs (V, G) that satisfy 
(|0.3|) . Theorem l4.5l states that the construction is a universal one. In Theorem 14. 10) 
we determine pairs which satisfy (|0.3|) and the following condition: 

(0.4) vectors s in (|0.2p are independent of v. 

The paper also contains a description of hulls Gv for G = Auto(-D), where D is a 
bounded symmetric domain in the canonical realization and Auto(-D) is the stable 
subgroup of zero, which coincides with the group of all linear automorphisms of 
D. These hulls have already been described: the final step was done in paper [9], 
which essentially used [10], partial results appear in [14] and [8]. Most of them 
use the technique of Jordan triples and Jordan algebras. We use Lie theory, in 
particular, an explicit construction of paper [15] for a maximal abelian subspace a. 
A compact group acting in a Euclidean space is called polar if there exists a subspace 
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(a Cartan subspace) such that each orbit meets it orthogonally. The group G is 
polar in the ambient linear space 0, and a is the Cartan subspace for G. Real polar 
representations are classified in paper J3J; they are orbit equivalent (i.e., have the 
same orbits) to isotropy representations of Riemannian symmetric spaces. If D is 
a polydisc B™ C C™, where D is the unit disc in C, then G = S n T n ; the polynomial 
hulls Gv are determined by the inequalities 

(0.5) (ik(z) < fik(v), 

where k = 1, . . . , n and are defined by 

(0.6) Hk(z) = max{|z CT(1 ) . . . z a ( k ) | : cr S S n }- 

The general case can be reduced to this one in the following way. Any bounded 
symmetric domain D C U of rank n admits an equivariant embedding of C" to 
0, which induces an embedding of B n to D, such that M" C C" is the maximal 
abelian subspace o, and, for any v S a , the hull of Auto(-D)v is the orbit of the hull 
of Auto(©™)i'. Each /ifc(z) has a unique continuation to a iT-invariant function on 
U. The extended functions determine hulls by the same inequalities. Moreover, they 
are plurisubharmonic and can be treated as products of singular values of z 6 D or 
as norms of exterior powers of adjoint operators in suitable spaces. The subsystem 
of long roots of the restricted root system (i.e., the root system for a) has type nA\; 
this defines the above embedding C" — * 0. Furthermore, this makes it possible to 
determine hulls in terms of the adjoint representation (Theorem 15. 7p . Thus, there 
is no need to consider different types of domains separately. 

The reduction to the case of a torus extended by a finite group, which is de- 
scribed above, is contained in Section [5] (in papers [9] , [T3] , the problem is also 
reduced to this case by another method). It does not use essentially the results 
of the previous sections (only Proposition 13 . 21 in proof of Theorem 15 . T[) . These 
extensions satisfy conditions (|0.3p and (|0.4| ; in addition, they possess the property 
that the complexified groups have open orbits. According to Theorem 14.101 any 
group with these properties is the product of groups S n T n acting in C n ; it admits 
a natural realization as a group of automorphisms of a bounded symmetric domain 
(Corollary [L3J. 

The following simple examples illustrate the case Gv % T c v and show that 
condition ()0.3|) is essential. Let G — S n T n , and let ei, . . . , e„ be the standard base 
in C". Then Gt\ is the closure the union of discs De^, k = 1, . . . ,n. Set H = S n T, 
where T acts by z — > e lt z, t € M, z € C". Then He\ = Ge\. For v = e\ + £2, Gv is 
the closure of the union of (™) bidiscs but T™ contains no proper torus T such that 
Gv = Hv for H = S n T. However, for any subgroup F C S n which acts transitively 
on 2-sets and H = FT n we have Gv = Hv. 

1. Preliminaries 

We keep the notation of Introduction, in particular, (jO.ip and (10. 6|) . Linear 
spaces are supposed to be finite dimensional and complex unless the contrary is 
explicitly stated. "Generic" means "in some open dense subset". Throughout the 
paper, we use the following notation: 

1D> and T are the open unit disc and the unit circle in C, respectively; 
V denotes a complex linear space (except for Section [5|); 
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if V is equipped with a linear base identifying it with C", then T™ is the 

group of all diagonal unitary transformations; 

Z£ consists of all transformations in T n with eigenvalues ±1; 

e\,. .. ,e n is the standard base in C™ and 

R" is the set of vectors in R™ with positive entries; 

Sk denotes the group of all permutations of a finite set K; if K = {1, . . . , n}, 
then Sk = S n ; 

C(K) is the algebra of all complex-valued functions on K; 
1 is the identity of C(K); 

G C GL(V) is a compact group whose identity component is a torus T 
(except for Section [5|); 

t C Ql(V) is the Lie algebra of T, t R = it,f = t+ t R ; 
T* = exp(t R ), T c = exp(t c ); 
C* = T c = C\ {0}; 

f = Hom(T, T) is the dual group to T; 

Aut(Z?) is the group of all holomorphic automorphisms of a domain D C V , 
Auto(-D) = Aut(L>) n GL(V); 

cone A denotes the least convex cone which contains the set X; 
convX is the convex hull of X; 
clos X is the closure of X; 

span F X is the linear span of X over the field F = C, M, Q. 
Clearly, exp is bijective on t R and T R = T c /T. The differentiating at the identity e 
defines an embedding of T into the dual space t*: x ~ > ~id e Xy where \ € T. This 
is a lattice in the vector group t*, moreover, T = t/L, where L is the dual lattice 
to f in t. For xef, let 

V x = {v € V : gv = x(g) v f° r all 9 G T} 
be the corresponding isotypical component of V. Then 

(i-i) ^ = E xef ©^- 

We assume that V is equipped with a G- invariant inner product ( , ). Then de- 
composition (jl.ip is orthogonal. Let spec(w) denote the spectrum of v € V (the set 
of x € T such that the x _ component of v is nonzero); for X <Z V, 

spec(A) = U xeX spec(x). 

We say that T has a simple spectrum if 

(1.2) dimV x <l 

for all x € T. If (| 1 . 2|) is true, then there exists a unique (up to scaling factors) 
orthogonal base in V which agree with (jl.ip and a unique maximal torus T™ in 
GL(V) which contains T. In what follows, we assume that (|1.2p holds; we shall see 
in the next section that such assumption is not restrictive. Thus, we may fix an 
identification 

(1.3) V = C n = C{K), 

where K = {1, . . . , n}. If F is a subgroup of Sk, then C(K) F denotes the set 
of all ^-invariant functions on K; clearly, 1 S C(K) F . Further, (C*) n is the 
multiplicative group of all invertible functions in C(K), T™ consists of functions 
with values in T, and (T") c = (C*) n . The Lie algebra of T" is realized as iW l C C n . 
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The embedding T — > T ra induces embeddings of the Lie algebra and the fundamental 
group: t -> iW\ m(T) -> iZ n C iE n , respectively. Let L be the image of tu(T). 
Then span R L = t; moreover, tniZ™ = T and t/L = T. The dual mapping f n -► f, 
which is defined by the restriction of characters e _ *' x ' 1 '' ) where x € iZ n , to t, is 
the orthogonal projection m : iZ™ — ► t. Thus, T is a subgroup of finite index in 
T = niil/ 1 . Vectors in span^ T are called rational. The image of t in iW 1 can be 
distinguished by linear equations with integer coefficients. Hence, clos(T c i>), for 
a generic v £ V, is the set of all solutions to a finite number of equalities with 
holomorphic monomials. Thus, Y C T c implies Y C clos(T c v). Set 

(1.4) c T = t K n closer;), 

The cone iC T is dual to cone(specy) Ct'C iW\ If -£ e clos(M^), then 4 = 
limt—j+oo exp(i£) is an idempotent in C(K) such that the multiplication by the 
complementary idempotent 1 — l is a projection onto span c (spec(£)). Set 

(1.5) It = {hm t ^ +00 exp(i£) : £ € C7 T }. 
Clearly, It is finite and contains 1. 

Lemma 1.1. The closure o/exp(Cr) is equal to It exp(Cr) ■ 

Proof. Due to the evident inclusion clos(exp(CT)) 3 /Texp(Cr), it is sufficient to 
prove that the set St = It exp(CT) is closed. Clearly, St is an abelian semigroup. 
The cone Ct is polyhedral; hence, it is finitely generated: 

C T = cone{£i, . . . ,£ m }, 

where R + £fc are the extreme rays of Ct, k = 1, . . . ,m. Obviously, It is a finite 
semigroup, which is generated by the idempotents lim t ^ +00 exp(i^). Thus, the cor- 
respondence (e - * 1 , . . . , e~ tm ) — ► exp(ii£i + . . . , t m ^ m ) defines a mapping of (0, l] m 
onto exp(C*r), which continuously extends to [0, l) m . It follows that its image is 
closed and coincides with St- O 

Note that there is a natural one-to-one correspondence between It and the set 
of faces of Ct- 

2. Hulls of finite unions of T-orbits in a T c -orbit 

Let v £ T c . If v = ^2 x£ f v x> wriere v x ^ ^x; 9 ^ an< ^ M = then 
M = S x ef X(g)v x - Since x(flO 7^ for all g £ G and \ £ f, we get 

(2.1) u e T c t; spec(u) = spec(i>); 

(2.2) dim (V x n span c Ti>) < 1 for all v £ V and y G T. 

Thus, the assumption that T has a simple spectrum in V is not restrictive in 
the problem of description of polynomial hulls of orbits Gv such that Gv C T c v. 
Clearly, D" = T™ in L(V). For each x £ Ct and any polynomial p on L(V), the 
holomorphic function /(C) = p(exp(C t x)) is bounded in the halfplane n : ReC > 0. 
Hence, exp(n) is contained in T. On the other hand, if z £ D™ n T c , then z = 
iexp(x) for some t £ T and x £ Ct (the polar decomposition). By Lemma 1 1.11 

f = clos(D n n T c ) = Tclos(exp(C T )) = TI T exp(C T )- 
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If v G (C*) n , then (C*) n v = (C*)™, and the mapping z — > zv is a linear nondegen- 
erate transformation of C™. Therefore, 

(2.3) ue(C*) n =S> fi = fu = T/ T exp(C T )w. 
For an arbitrary v G V — C™, set 

= U G t R : 6 < if v k ? 0, k = 1,. . .,n}. 
Applying (12. 3p to span c (spec(i>)) = C™v, we get 

(2.4) fv = Tclos(exp(C^>. 

Clearly, C£ depends only on spec(v). For s G 1" and z G (C*) n , set 

If Sk > 0, then the k-th factor in (C*) n can be replaced with C (i.e., v s continuously 
extends to this product). 

It is well known that for any holomorphically convex T-invariant set U C T c , 
the set log(J7 D T R ) C t K is convex. In particular, this is true for sets of g e T c 
such that gv £ TX, where X C T c v, v £ V . Nevertheless, it is convenient to have 
an explicit construction of an analytic strip (or an annulus, if it is periodic) in a 
T c -orbit, which corresponds to a segment that joins two points in t K ; it is contained 
in the following lemma. Set 

S ={zeC: <1mz < 1}. 

Lemma 2.1. Let v e C" and u G T m v. Then, there exists £ G t K such that 

(2.5) A(z) = exp(z£)v 

is a holomorphic mapping X : S — > T c v which satisfies conditions 

X(dS) CTuU Tu, 
A(0)=k, A(l)=u. 
// t/ie stable subgroup of v in T K is trivial, then £ is unique. 

Proof. These properties hold for £ G t R such that exp(£)u = u; such a £ exists, since 
exp is a bijection t M — ► 7*. The last assertion is clear. □ 

If £ G C't, then (|2.5|) defines an analytic halfplane in Tv; for T-rational £, A is 
periodic and defines an analytic disc in Tv. Together with Lemma 12. II this gives a 
characterization of hulls for finite unions of T-orbits in T c . Suppose that X c T M v 
is finite and the stable subgroup of v in T is trivial. Then, the inverse to the 
mapping x — > exp(x)i> is well defined. Let us denote it by log„ and set 

(2.6) Q x =conv(lo g!J X), 

(2.7) P x = Qx + C T - 

The set Px is a convex polyhedron, which is unbounded if Ct ^ 0. Hence, there 
exists a finite set 9Tx C M. n and, for each s G O^x, real numbers c s such that 

(2.8) P x = {x G t R : (at, s) < c s for all s G 

The set Vlx consists of vectors orthogonal to faces of Px, whose projections into 
span K Px look outside of it; clearly, it is not unique in general. 
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Proposition 2.2. Let v G (C*) n . Suppose that Y C T c v is a finite union of 
T -orbits (including Tv), and set X = T K v (1 Y. Then X is finite and 

(2.9) Y = clos(Texp(P x )u) 

(2.10) = clos{z G (C*) n : v„{z) < e c >v s {v), s G m x } 

(2- 11 ) = U P ,0,^ 

(2.12) = Texp(P x )I T v, 

where Qx , Px > are as above and It is defined in (|1.5[) . 

Proof. Due to the polar decomposition, the set TuC\T R v, for each u G T c v, is non- 
void and consists of a single point. Hence, X is finite and Y = TX. The inclusion 
(e-xjpQx)v C Y follows from Lemma 12 . 1 1 and Phragmcn-Lindelof Principle. The in- 
clusion exp(Cx)u C Tu is true for any it € C". Since it holds for all u G Texp(Qjs:), 
the left-hand side of (|2.9j) includes the right-hand side. Hz — exp(^)w, where ^ G t c , 
then Zfc = e^ k Vk, k = 1, . . . ,n; due to (|2.8[) . this implies that the right-hand side 
of ([23)1 coincides with (|2TTD|) . According to ([23)1 . the right-hand side of (|23)l and 
((2~TT|l intersect T c w by the set 

Texp(P x )v = exp(Q Jf )rexp(C T )w; 

clearly, it is dense in (12. lip . Since Qx is compact, the set (|2.11|) is closed. The 
compactness of Qx, the above equality, and Lemma ITTTI imply that (|2.12p is closed; 
hence, it is the same as the right-hand side of (12. 9[) . 

Each of the sets (|2~9| - (|2~T2|) includes Y. Thus, it remains to prove that (f27T2|) 
is polynomially convex. If x E t R \ Px , then there exists s G t™ such that 

(2.13) sup{(y,s):yePx}<(x,s). 

Since Qx is compact, the linear functional on I in the right-hand side of (|2.13p must 
be nonnegative on Ct- According to (|1.4p . we may assume that s G closM™. It 
follows that (|2. 13jl holds in a neighborhood of s in closlR™ . Thus, s can be assumed 
rational (hence, integer) with strictly positive entries. Then, p(z) = z^ 1 . . . z„ n is 
a holomorphic polynomial such that \p \ separates exp(x)w and Tclos(exp(Px)u). 
Therefore, 

?nT c v = Texp(P x )v. 

For any t G It, the projection z — > lz commutes with T. This makes it possible to 
apply the above arguments to the vector uv, the set iX, and to the restriction of T 
to iC n . Consequently, 

(2.14) lY DT c lv = Texp(P tX )iw = iT exp(P x )v 

(clearly, iexp(P x )v — exp(P L x)iv). By (|1.5p . CY C Y, hence, CY C Y"; on the other 
hand, lY C since poi is a polynomial on C™ for any polynomial p on tC n . Thus, 
<,y = = Y" n tC" . Together with (|2.14p , this implies the polynomial convexity of 
(l2~T2l) . □ 

If T = T™, then Proposition 12 . 21 follows from the well-known characterization of 
polynomially convex Reinhardt domains. 

Corollary 2.3. For any v G (C*) n } the orbit T c v is closed in C™ if and only if Tv 
is polynomially convex, and this is equivalent to Ct — 0. Then, Y — Texp(Qx)v 
for all y, X as above. 
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Proof. The orbit T c v is closed if and only if the convex hull of spec(f) = spec(C") 
contains in its relative interior (see, for example, |13[ Proposition 6.15]). Since 
T C GL(n, C), the set spec(C") is generating in t*. Hence, T c v is closed if and 
only if C T = 0; by ([2T4]) . this is equivalent to Tv = Tv. Then, Y = Texp(Q x )v by 
([HID and (EH). □ 

There is a version of the first assertion for an arbitrary compact linear group G: 
a G c -orbit is closed if and only if it contains a polynomially convex G-orbit ([Jj 
Theorem 1 and Theorem 5]). For a torus T, all T-orbits in T c v are simultaneously 
polynomially convex or non-convex, but this is not true if G is not abclian. 

3. Finite extensions of T that keep a T c -orbit 

In this section, we consider the case where the set X defined in the previous 
section is an orbit of a finite group F which normalizes T and keeps the T c -orbit. 
We assume that T C G, T is a torus, G is a subgroup of GL(V), F is a finite 
subgroup of G, and 

(3.1) G = FT = TF, F = G/T, 

(3.2) Gv C T c v, 

(3.3) v e (C*) n c C" = V. 

By (j3Tj) . T is normal in G. Clearly, (|3~2|) is equivalent to Fv C T c v and to the 
connectedness of G c . Here is an illustrating example. 

Example 3.1. Let G = Aut (ID> 2 ) be the group of linear automorphisms of the 
bidisc D 2 C C 2 . Clearly, G = FT, where F = S2 is generated by the transposition 
r of the coordinates, T = T 2 , T c = (C*) 2 , and T c v = (C*) 2 for any v that lies 
outside the coordinate lines. Thus, (|3.2| holds for all v £ (C*) 2 (however, (|3.2| fails 
for any v ^ in C 2 \ (C*) 2 ). The hull Gu can be distinguished by the inequalities 

(3.4) max{|zi|, \z 2 \} < max{\vi\, \v 2 \}, 

(3.5) \ziz 2 \ < \viv 2 \. 

Clearly, (|3.4p and (|3.5p define a polynomially convex set. Let zi,Z2 > (a generic 
T-orbit evidently contains such a point z). Then, z and rz can be joined by an 
analytic strip with the boundary in Tz U Ttz: 

X z {s) = {z\- S z s 2 ,z{z\- S ), seS. 

Set q = In and let zi > z 2 ■ Then, the strip can be written in the form 

A z (s) = (e _s zi, e s z 2 ), 0<Res<q. 

It is periodic with the period 2ni and defines a r-invariant annulus in Gv with r- 
fixed points (^/ziZ2, -y/zlii) an< i ( — -\fzxZ2, —^Jz\z 2 ). As Z2 — > 0, the annulus tends 
to a couple of discs: (e _s zi,0) and (0, e~ s zi), where Res > 0, < Ims < 27r (the 
circle Re s = | , < Im s < 27r collapses to zero) . Let z S Gw n M 2 . Then Gi> 
contains a bidisc D 2 z. It intersects K 2 by a rectangle, which is symmetric with 
respect to the coordinate axes. If z lies on an axis, then the rectangle degenerates 
into a segment. Let vi > v 2 > 0. The union of these rectangles with vertices 
in the set Q of real points of the annulus, which joins v and tv, is a curvilinear 
octagon. It degenerates into a pair of segments if v 2 = and into a square if v\ = v 2 
(see jTQl Fig. 2] for the 3-dimensional case). In the logarithmic coordinates in the 
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first quadrant, Q is a segment. Also, note that all nontrivial T -orbits are not 
closed. □ 

In [2], Bjork found a typical situation where analytic annuli appear in the maxi- 
mal ideal space Ma of a commutative Banach algebra A which admits a nontrivial 
action of T by automorphisms: this happens if T-invariant functions on Ma do 
not separate distinct T-orbits. In [7], it was noted that analytic strips and/or an- 
nuli appear in Gv if the stable subgroup of v in G does not coincide with the 
complexification of the stable subgroup of v in G. 

Proposition 3.2. The hulls Gv for orbits of G = Auto(B rl ) = S n T n are distin- 
guished by inequalities (|0.5p . where fj>k are defined by (|0.6j) . 

Proof. The approximation by decreasing sequences of hulls makes it possible to 
reduce the proposition to the case of a generic v in (|0.5|) . Then, applying to v = 
(vi, . . . , v n ) a suitable transformation in T", we may assume that 

(3.6) vi > v 2 > ■ ■ ■ > v„ > 0. 

Moreover, we may use Proposition 12.21 with X = S n v, Ct = — closR" (we keep 
the notation of Proposition 1 2 . 2|) . Since X, Qx, Ct, Px-i an( i Mfe are fi^-invariant, 
S n is transitive on X, by ()0.6|) . (|0.5p . and (|2.10p . it is sufficient to prove that the 
vectors = 22 r =i £r ' ^ ~ I, ■ . ■ ,n, correspond to the faces of Px that meet at v, 
are orthogonal to them, and look outside of Px ■ 

Set 77i = e 2 - ei, . . . , 77„_i = e„ - e n _i,77„ = -e„. Then {-n k }l =1 is a base in 
R n , which is dual to the base {£,k}k=i- We claim that the cone of the polyhedron 
Px at the vertex v is generated by {r]k}k=i- This implies the assertion above 
(note that both cones are simplicial). If r S S n is a transposition (k,j), then 
v — tv = (vk — vj)(ek — If ct, k G S n then v — onv = (v — kv) + [kv — ctkv). 
Furthermore, S n is generated by transpositions (k, k + 1), and Vk — ffe+i > by 
(|3.6p . where fc = 1, . . . , n — 1. Therefore, vectors 771, . . . , 7] n -i generate the cone of 
Qx at v. Since — ej, = X)r=o ^"-^ anc i Ct is generated by — efc, A; = 1, . . . ,n, this 
proves the proposition. □ 

Property (|3.2p implies span c Gu = span c T c w. Hence, we may assume that (|1.2p 
is valid. Then, a generic £ G t has a simple spectrum. Any / G F permutes 
eigenvalues and eigenspaces. Thus, assuming (|1.2p and identifying V with C(K) 
in accordance with (| 1 .3[> . we get that each element of F is a composition of a 
permutation of X and a multiplication by a function on K. Further, (|3 . 3[) implies 
that the stable subgroup of v in T c is trivial. Hence, 

T R v = T c /T == t R , 

where the identification of T R w and t R is realized by ^ — ► exp(£)v, ^ G t R . 

Lemma 3.3. Let G C GL(V), a subgroup F C G, and v G V satisfy \3.1\) 
A3.3\) . Then T c v contains a G-invariant T -orbit. Moreover, there exists a mapping 
f — > tf , F — > 7 1 smc/i t/iot F — {tf f : f G -F} is a subgroup of G which has a fixed 
point in T c and satisfies h3.1\) ~ \3.3\) . 

Proof. The group i 7, naturally acts on T R u = T c /T = t R . Any g G F is a composi- 
tion of ct G and a multiplication by a function in C{K). Since t acts on C(K) 
by multiplication on linear functions and ct induces a linear transformation in t , 
the induced action of F on t R is affine. Since F is finite, it has a fixed point in t R . 
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Hence, T c v contains a G-invariant T-orbit. Let us fix a point u in it and define 
tf by tffu — u; the choice is unique due to (13. 3p . Taken together with ()3.1|) . this 
implies that F is a group, which obviously satisfies the lemma. □ 



According to Lemma 13.31 we may assume without loss of generality that 

(3.7) fv = v for all / e F. 

In the following example we give a construction (associated with a given finite group 
F) for orbits with property (|3.2[) . 

Example 3.4. Let t be a real linear space, t* be the dual space to t, L be a lattice 
in t, and L* C t* be the dual lattice to L. Set 

K(y) = y(x), where x € t, yet*. 

Let K be a finite subset of L* that generates L* as a subgroup of the vector group 
t*. Then 

(3.8) t* = span R K, 

(3.9) L = {x e t : X X {K) C Z}. 

Further, let F be a finite subgroup of GL(t) which keeps K. Set V = C(K). The 
mapping 

(3.10) A : x — > iAz | „ 

is an embedding t — > V, which has a natural extension to t c . Set 

(3.11) exp(x) = e 2 * iX *. 

Clearly, L = kerexp. Hence, exp defines an embedding of T = t/L and T c into the 
group (C*)": 

T c = exp(t c ) C (C*) n . 

The group T c acts on C(K) by multiplication. The inclusion v £ C(K) F is the 
same as (|3.7j) : it implies (J3T2J) . Furthermore, if u € (C*) n , then 

(3.12) span c Tu = y. 

Indeed, the space span c T is a subalgebra of C(K), which separates points of the 
finite set K. Hence, it coincides with C(K). □ 

Theorem 3.5. Let a group G C GL(V^), a finite subgroup F C G, a torus T , and 
a vector v € V satisfy (TO p -jRO )) , pTTp , anrf gjj) . Tften V,G,F,T,v can be 
realized as in Example \3.4\ where 

(3.13) v e (C*) n nC(K) F . 

Conversely, ifV,G,F,T,v are as in Example \3.4\ and v satisfies H3.13\l , then \3. 1\) 
< TOI) . (3~7\ ), and i TOl) are true. 

Proof. The group F acts in t and t* by the adjoint action. Let if C t* be the 
collection of all weights for the representation of T in V; clearly, K is F- invariant. 
It follows from (|3.12p and (|3.2p that the weights are multiplicity free. This defines 
an equivariant linear isomorphism between V and C(K), where the group T acts by 
multiplication. Thus, A and exp are well defined by (|3.10|) and (|3. 1 1|) . According 
to (|3.7| and (|3.3| . (|3.13p is true; (|3.8p holds since T C GL(V) is compact and acts 
effectively on V (note that the annihilator of span R K in t acts trivially due to 
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(pUl) and (f37T3]> ). Let us define L by (JUl). Then L = kcrexp by (pUT|) . Hence, L 
is a lattice in t and the group L* generated by K is the dual lattice in t*. 

The converse was proved in Example 13.41 □ 



4. Finite extensions of T which keep generic T c -orbits 



In what follows, we use the setting of Example 13.41 Let Z denote the centralizer 
of G in GL(V). We assume that (C*) n acts in V = C(K) by multiplication. 

Lemma 4.1. Z = C{K) F n (C*) n . 

Proof. Since A(t) separates points of K, Z C (C*) n . The multiplication by u <E 
C(K) commutes with F if and only if u is ^-invariant. □ 

In general, condition (|3.2p does not hold for a generic vector v. Hence, there is 
a natural problem: describe V and G such that generic orbits satisfy if3.2|) . The 
following proposition contains a simple criterion. 



Proposition 4.2. Let V, G be as in Example \3.4\ Then G satisfies S3. 2\) for a 
generic v € V if and only if 

(4.1) C(K) = \(t c ) + C(K) F . 

In this case, each T c -orbit in (C*) n intersects C(K) F . 

Proof. It follows from Lemma [4.11 that the right-hand side of (|4.ip is the tangent 
space at 1 to the set T c Z. Clearly, G c = ZG C is a group, T C Z is the identity 
component of G c , and the right-hand side of (|4.ip is the tangent space to G c l. 
Hence, (14.1[) holds if and only if G c l is open. Moreover, this is equivalent to the 
equality T C Z = exp(A(t c ) + C{K) F ) = (C*) n . Therefore, each T c -orbit in (C*) n 
intersects C(K) F , i.e., contains an F-fixed point. Thus, (|4.1|1 implies (|3.2p for 

v e (C*) n . 

Let (|3.2p hold and let W be an F-invariant neighborhood of 1. If is sufficiently 
small, then the condition log 1 = defines a branch of log in W . We may assume 
that log W is convex and symmetric. This makes it possible to define roots in W: 

wr = cxp (i logw). For v € W% and / <E F, set gf = f^pj ^ > where r = cardF, 
and g = Ylfepdf- Then gv is F-fixed. If (|3.2p holds for v, then gf S T c for all 
/ e F; hence, gv £ T c v. Consequently, for all v € W, T c v intersects C{K) F . Since 
Z keeps this property of orbits, it follows that T C Z has a nonempty interior. This 
implies (|Q|) . □ 



Theorem 4.3. Let G C GL(V) be a semidirect product of a torus T and a finite 
subgroup F , and let Z be the centralizer ofG in GL(V). Suppose that span c Tv = V 
for some v G V. Then the following conditions are equivalent: 

(i) Gv C T c v for a generic v G V ; 

(ii) G Zv is open in V for a generic V € V. 



Proof. By Theorem l3.5l we may use the construction of Example [331 According to 
Lemma l4~Tl (ii) is equivalent to (|4.1|1 , and the assertion follows from Proposition ^. 21 

□ 
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We shall give a constructive description of these spaces and groups. Set 

C (K) = {ue C{K) : J2 qeK <<D = °} • 

Sometimes, we identify points in K with their characteristic functions. 

Example 4.4. Let V = C n = C(K), where K = {1, . . . , n}, let F be a subgroup 
of S n , and 

(4.2) K = KiU---UK p 

be the partition of K into F-orbits. For k £ {1, . . . ,p}, set V% = C(K k ). Then 
V = Vx ® ■ ■ ■ ® V p . Set 

t° k = c (K k )mw\ 

T° = exp(t°) c C(K k ), 
where exp is defined by (|3~TTj) . Set t° = t? 8 • • • © t°, 
T° = exp(t°) = I? x • • • x T°. 
Let T be an .F-invariant torus such that 

(4.3) T° C T C T" 

and set G = FT. Then generic G c -orbits satisfy (|3.2p . The group G is irreducible 
if and only if F is transitive on K\ in general, F-orbits in K define G-irreducible 
components of V. There are two extreme cases in (|4.3|) . 

(A) If T = TP, then there is one open orbit (C*)" of the group G c = FT C , which 
evidently satisfies (|3.2p . If F is nontrivial, then there exist degenerate orbits 
that do not satisfy (|3.2[) ; moreover, if F is transitive on K, then all non-open 
G c -orbits, except for zero, are nontrivial finite unions of T c -orbits. 

(B) If T = T , then generic orbits are closed. They have codimension p and 
are distinguished by equations 

LI ^ is Z r = c k, 

where c k 6 C*, k = 1, . . . ,p. 

Note that (A) and (B) are invariant under the Cartesian product (the group F 
need not be the product of groups Fu of irreducible components but must have the 
same orbits in K as F\ x • • • x F p ). In terms of Example 13.41 in (A), t = M™, the 
mapping A : t c -> C(K) is surjective, K = {e±, e„}; in (B), t = iR™ n C {K), 
A(t ) = Cq(K), and the set K is the projection of {ei, . . . , e„} into t* = t. In both 
cases, K is the set of all vertices of a regular simplex. □ 

Theorem 4.5. Let V,G be as in Theorem \4-3\ and let (i) hold. Then V,G can be 
realized as in Example \4--4\ Furthermore, 

(1) V, G are of type (A) if and only if G c has an open orbit, 

(2) (B) is equivalent to the assumption that the center of G is finite, 

(3) if G is irreducible, then either (A) or (B) holds. 

Let C(K)^_ be the cone of all nonnegative functions in C(K) F . 

Lemma 4.6. Let G and V be as in Example \3.4\ Then, the orbit G c v is closed 
for a generic v £ V if and only if 

(4.4) t R n C{K)l = 0. 



ORBITS OF TORI EXTENDED BY FINITE GROUPS AND THEIR POLYNOMIAL HULLS: THE CASE OF CONNECTED COMP 



Proof. Clearly, G c v is closed if and only if T c v is closed. Let v G (C*) n . By 
Proposition 12.21 and Corollary 12. 3[ T c v is not closed if and only if Ct 7^ 0. Since 
Ct is -F-invariant by (|1.4[) . it contains YlfeF f u ^ or eacn u e Thus, Ct = is 
equivalent to (|4.4p . □ 

Proof of Theorem \4-.5\ Suppose that G is irreducible or, equivalently, F is transi- 
tive. Then Z = C* 1 according to Lemma O If 1 G A(t c ), then T c D Z and 
T c i> is open for a generic w G V by Theorem 14.31 If 1 ^ A(t c ), then (|4.4p is true; 
by Lemma [4.61 T c i> is closed for a generic u£ V, By Proposition 14.21 a generic 
T-orbit intersects C*l. Consequently, we have 

(4.5) codimG c w = l. 

Let 1 G T c n Z. The orthogonal projection of 1 into the tangent space TiT^l is 
F-fixed. Hence, it is proportional to 1; since 1 ^ A(t c ), this implies 1 _L A(t c )l. 
Therefore, TiT c l coincides with the tangent space to the hypersurface z\ . . . z n = 1 
at 1; since the monomial on the left is an eigenfunction of T c , this group keeps it. 
Due to (|4~5jl . T c l coincides with this hypersurface. Then, T = T n n SU(ra), and 
any T-orbit that intersects Z is a hypersurface Zi . . . z n = c, for some c G C* . This 
implies t c = C Q (K) and T = T°. 

Thus, the theorem is proved for all irreducible G. The projection onto each 
irreducible component keeps the property (|3.2p for generic orbits since it commutes 
with G. Hence, (i) holds for all irreducible components. They correspond to F- 
orbits Kk in the partition (|4.2p . Let t2, k = 1, . . . ,p , be defined as in Example 14.41 
According to the arguments above, X(t\x k ) 3 A(i2) for all fc. If x G t, then the 
averaging 

Ax = ±Y,feFf x ' r = caxdF, 

distinguishes the F-fixed component of x (i.e., Ax G C(K) F n t and x — Ax G t°); 
since t is F-invariant, it contains both components. By Lemma l4.1i if G has a finite 
center, then A (t|jf fc ) = A(t°) for all k. It follows that 

t C t° = t? 8 • • • ® t°. 

On the other hand, (ii) and Lemma H. II imply codimt < dimC(K) F = p. Hence, 
the inclusion above is in fact the equality. Thus, we get (B) assuming that G has 
a finite center. The converse is true since t does not contain a nontrivial F-fixed 
element. The same arguments show that any F-invariant torus T includes T° if (i) 
is true. This proves that V, G admit the realization of Example 14.41 (1) and (2) are 
clear. □ 

Corollary 4.7. Let G be as in Theorems 4-5 and 4-3- Then G contains a closed 
subgroup G° such that 

(1) each connected component of G contains a connected component of G° , 

(2) G° has a finite center, 

(3) generic orbits of (G°) c are closed, 

(4) Gv n T R v = G°v n {T°) R v for a generic v G V. 

Proof. By Theorem S3] and g^J), G D T°, where T° is as in (B). Clearly, F 
normalizes T°. Hence, G° = FT is a group, which satisfies the corollary. □ 
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Proposition 12.21 makes it possible to find Gv for G as above. If T = T™, then 
T Z) and generic T-orbits intersect R"; hence, we may assume v £ K™. Then 
Twnl" is a parallelepiped 11^ = conv{(±Wi, . . . , ±v n )}. Clearly, = Z 2 l H+ , 
where 11+ = n„ n closR™. Since R™ = T m v, we may use Proposition 12.21 with 
X = Fv, C T = - closR^, P x = conv(Fv) - Rp 

Gv = U uecxp (Q tj )D™u = rU„ erap (Q t ) U u = TUugoxpfQ^) n + , 

where Q v — conv_Fu. For the description in the form (|2.10|) , one has to know 
normal vectors to faces of conv_Fu. Since F may be an arbitrary subgroup of 
S n , they need not be proportional to rational vectors (for example, this is true 
for the cyclic subgroup of order 3 in S3). We shall describe the situation where 
they are locally independent of v; since they depend on v continuously, this is 
equivalent to the condition that they are rational. Note that the vector which joins 
two points in t K as in Lemma 12.11 is rational if and only if the strip reduces to an 
annulus. In Example 14. A\ F need not be the product of groups corresponding to 
the irreducible components; we shall see that F possesses this property in the case 
under consideration. 

Let U be a real vector space and F c GL(J7) be a finite group. Set 

C u — cone(w — Fu); 

this is the cone at the vertex u of the polytope conv(_F'u) (which may be degenerate). 
We say that C u is locally independent of u if, for a generic u £ U , C u = C w for all 
w that are sufficiently close to u. 

Lemma 4.8. Let U be a real vector space and F be a finite subgroup of GL({7). 
Suppose that C u is locally independent of u. Then F is generated by reflections in 
hyperplanes in U . 

Proof. We may assume without loss of generality that U is equipped with an inner 
product and that F C 0(U). Let R+(it — fu), f £ F, be an extreme ray of C u . 
The equality C u = C w for w in a neighborhood of u implies that this ray does not 
change near u. Hence, dim(l — f)U = 1. Since / is orthogonal and nontrivial, it 
is a refection in a hypcrplane. The stable subgroup of a generic u £ U is trivial 
(hence, F acts freely on a generic orbit) and each vertex of conv^it) can be joined 
with u by a chain of edges. Applying the above arguments repeatedly to u, fu, etc., 
we get that F is generated by reflections in hyperplanes. □ 

For any g £ T^S n and k = 1, . . . , n, gek — ± e a ^) for some a £ S n . The mapping 
/ — > a is a natural homomorphism 1QS n — > S n , which we denote by (j). 

Lemma 4.9. Let F be a transitive subgroup of S n acting in R" by permutations 
of coordinates and let a group H C Z^Sn be generated by reflections in hyperplanes 
in W l . Lf 0(H) = F, then F = S n . 

Proof. Let p be a reflection in a hyperplane in R™. If p £ 1^S n — BC n , then it 
is conjugate to a reflection in a wall of the Weyl chamber that is distinguished by 
the inequalities x\ > ■ ■ ■ > x n > 0. Hence, 4>{p) is a transposition if it is nontrivial. 
Since F = <f>(H), F is generated by transpositions. It remains to note that any 
subgroup of S n , which is generated by transpositions, coincides with S n if it is 
transitive on {1, . . . , n} (consider the graph with the vertices {1, . . . , n} and edges 
corresponding to transpositions and note that inclusions (k,l) £ F, (l,m) £ F 
imply (k, m) £ F; this makes it possible to use the induction). □ 
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We say that a pair (V, G) is standard if it is isomorphic to (A) or (B) in Ex- 
ample [474] with F = Sk- The product of pairs (Vk,Gk), k = 1,. ..,m, is the pair 

(ELi^ITZLiG*). 

Theorem 4.10. Let G = FT be a compact subgroup of GL(n, C), where T C T™ 
is a forus and F is a subgroup of S n . Suppose that Gv C T c v for a generic v & V 
and 

(1) either T = T" or t/ie center of G is finite, 

(2) for a generic v £ C n , Gv can be distinguished in closT c w by a family of 
inequalities 

\zx\ s K..\z n \ s - <p s (v), 

where p s {v) > and vector s = (si, . . . , s n ) runs over a certain finite subset 

o/R n which is independent of v. 
Then (V, G) is isomorphic to the product of standard pairs. Moreover, if G is 
irreducible, then (V, G) is standard. 

Proof. Let G be irreducible. Then F is transitive and (V, G) are as in (A) or as 
in (B) by Theorem 14.51 Suppose that (B) is the case. It follows from (2) and 
Proposition 12.21 that the polytope Qx C t R , where X = Gv D T R v, for a generic 
v, satisfies the assumption of Lemma H. 81 Therefore, F is generated by reflections 
(we may assume that F C 0(t R )). They extend to reflections in hyperplanes in 
t 8 + Rl = M™ if we assume that they fix 1. Then, Lemma EM implies F = S n . The 
case (A) can be reduced to (B): it is sufficient to replace T™ with T — SU(ra) n T" 
since F evidently keeps T and to note that (2) remains true due to Proposition ^. 21 
Thus, (V, G) is standard. 

Let the center of G be finite. According to Theorem 14. 5( T may be identified 
with the group T° in Example 14.41 In particular, G c v is closed for a generic v 
and Ct — due to Proposition 12.21 By Proposition 14.21 generic orbits contain 
F-fixed points. Applying the arguments above (which did not use the assumption 
that G is irreducible) , we get that the cones at the vertices of the convex polytope 
Qx, X = Gv n T R C t R , are locally independent of v. Clearly, the same is true 
for its projection into each space t° corresponding to an irreducible component 
14 of V = C". This implies that all irreducible components are standard. Thus, 
Fk = S(Kk), where k = 1, . . . ,p and K = K\ U • • • U K p is the partition of K into 
-F-orbits. Due to Theorem 14. 51 it is sufficient to prove that 

(4.6) F = F t x • • • x F p . 

By Lemma 14.81 F\ t o is generated by reflections in hyperplanes in t ; the condition 
that they keep real F-invariant functions on K uniquely defines their extension to 
W 1 . Hence, F is generated by reflections in M". A permutation which induces a 
reflection in a hypcrplanc in R" is a transposition of a pair of coordinates; this pair 
is necessarily contained in only one of the sets K^, k — 1, . ..,£>. This proves (|4.6[) . 

If T = T", then T is a product of tori in irreducible components. Thus, the case 
T = T" follows from the above case, since the assumptions of the theorem hold 
true for the group T° if they hold for T in (|4.3[) in Example 14.41 □ 

5. Hulls of isotropy orbits of bounded symmetric domains 

We start with a preliminary material on hermitian symmetric spaces following 
[15] but adapting the exposition to our purpose in order to be as self contained as 
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possible. For a subset X of a Lie algebra g, l{X) = {z £ g : [z,X] = 0} is the 
centralizer of X. Let G be a simple real noncompact Lie group with a finite center, 
K be its maximal compact subgroup, and g,t be their Lie algebras, respectively. 
If the center 3 = 3(6) of 6 is nontrivial, then g is called hermitian. Then £ = 3(3) 
and dim 3 = 1 (note that K is irreducible in Let c be a Cartan subalgebra 

of t. Then c is also a Cartan subalgebra of q and 3 C c. There exists k £ 3 such 
that ad(k) has eigenvalues Q,±i (it is unique up to a sign; kerad(k) = 6). Then 
k = e n&d ^ is the Cartan involution which defines the Cartan decomposition 

(5.1) = 60 0, 

where 6,0 are eigenspaces for 1,-1, respectively. Furthermore, j = ad(k) is a 
complex structure in 0. This defines the structure of a hermitian symmetric space of 
noncompact type in D = G/K. These spaces can be realized as bounded symmetric 
domains in C™ with K = Auto (I?). Any irreducible bounded symmetric domain 
admits such a realization. Let A C ic* be the root system of g c . Each a £ A 
corresponds to an sL^-triple h Q ,e Q ,f Q such that ib a £ c. Thus, a(b a ) — 2, [e Q ,f Q ] = 
h Q , and 

(5.2) [h,e a ] = a{h)e a , [ft,f Q ] = -a(h)f a 

for all h £ c c . We identify c c and (c*) c equipping q with an Ad(i^)-invariant 
sesquilinear inner product and normalize it by the condition 

(5.3) max{|a| : a £ A} = V2. 

Then short roots must have length 1 (note that G2 has no real hermitian form). 
The set A v = {h a : a £ A} is the dual root system. The above normalization 
implies h a = a for long roots and h a = 2a for short ones. Since ad(/i), h £ c, has 
eigenvalues and a(h), where a £ A, we get a(ik) = 0, ±1, i.e., ik is a microweight 
(of A v ). For s = 0,±1, set 

(5.4) A s = {a £ A : a(ik) = s}. 

Since 6 © ti) is a compact real form of g c and span R {ih Q , e a — f a ,i(e a + f a )} is the 
su(2)-subalgebra corresponding to a root a £ A, we have 

(5.5) d = span R {e a + f Q , i(e a - f Q ) : a £ Ai}. 

Set s a — span R {ih a ,e Q + f a ,i(e a — f a )}. Then s a is an sl(2, R)-subalgebra of g c 
and 

(5.6) a £ A ± i ^> s a C q. 

Let E be a maximal subset of pairwise orthogonal long roots in Ai. Set 

Let a, [3 £ E, a ^ [3. Since a, /3 are long and orthogonal, ±a ± /3 <^ A. Hence, 



(5.7) a,/Je£,a^ =► [s Q , S/3 ] = 0. 

ft follows that h, e, f is an s^-triple and s is a subalgebra of g. Set 

(5.8) 6» = e l^ad(e-f) 5 

(5.9) a = span R 



ORBITS OF TORI EXTENDED BY FINITE GROUPS AND THEIR POLYNOMIAL HULLS: THE CASE OF CONNECTED COMP 



Here is the standard realization of root systems B n and C n : 

B n = {±e k ± e/, ±e m : k,l,m = 1, . . . ,n, k < I}; 

C n = {±e k ± e/, ±2e m : k, I, m = 1, . . . , n, k < I}. 

Then C n = B^, but C n does not satisfy (|5.3|) . These systems have microweights; 
up to the action of the Weyl group, they are: 

B n : ei; 

C n : |(ei + --- + e„). 

There are no other irreducible root systems which have microweights and contain 
roots of different lengths. Also, B n and C n have the same Weyl group BC n — Z^Sn- 

Lemma 5.1. The space a is a maximal abelian subspace ofd. 

Proof. A straightforward calculation with 2-matrices shows that 8h 
(EH), 

(5.10) 9h a =e a +f a for all a E E. 

It follows from (|5.5p that a C 0. Moreover, a is abelian due to (|5.7p . Set 
We claim that 

(5.11) SCA . 

Indeed, a root in Ai n 5 must be short. This may happen only in B n or C n , since 
Gi and -F4 have no microweights and other irreducible root systems have no roots 
of different lengths. In B n , k is a short root and all other short roots are orthogonal 
to k. Hence, they do not belong to Ai. In C n , E = {2ei, . . . 2e„}; then S = 0. 
Since A_i = — Ai, this proves (|5.1ip . 

Set b = E n c and m = span c {e a ,f a : a 6 S}. It follows from (|5.1ip that 
m C t c . Clearly, %(E) = c c © m. The space m is ^-invariant, because 9 fixes roots 
in S. Due to (|5.9|) . we get 

3(a) = 03(£O = b c 0a c ®m 

Since b c m C t c , this implies 3(a) H 5 = 0. □ 

The projection of OA into a is the restricted root system A a (it is also the set of 
roots for ad (a) in g). The group 

W = {Ad(g) : geK, Ad( ff )a = a}| a , 

acting in a, is the Weyl group of a. 

In what follows, we denote by the complexification of a with respect to the 
complex structure j (thus, C D). The set BE is a base in 0; enumerating it, we 
identify with C™. Set t = span R iS, T = expt, H = WT. The torus T = T™ is a 
maximal compact subgroup in the group exps C G. 

Proposition 5.2. The following assertions hold: 

(1) A a is a root system of type BC n or G n ; 

(2) the pair (0, H) is standard with T = T™. 

Proof. (1). Let A a \ 8E contain a long root a. Then a = h(ot\ + a 2 + + 
04) for some ai,...,ct4 € 0.E, since |a| 2 = 2 and (a,/3) = 0,±1 for all /3 € E 
due to the normalization (|5.3p (note that a,/3 generate ^2 if («,/3) 7^ 0). Roots 
a, ai,..., 0:4 generate D4, since only A4 and I?4 among irreducible systems of rank 



= e + f . By 
5 = An£^. 
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4 consist of roots of equal length, but A4 does not contain an orthogonal base. Since 
(ik, 0) = 1 for all j3 € E, the projection of iOk into span R £>4 is a microweight u> 
such that (w, au) = 1, k = 1, . . . , 4, but -D4 has no microweight with this property 
(in the realization above, -D4 = -B4 n C4 and the microweights are either ie^ or 
^(±ei ± e2 ± e 3 ± £4)). Thus, EU (-E) is the set of all long roots in A a . According 
to the classification of irreducible root systems, only C n and BC n — B n U C n has 
the property that linearly independent long roots are mutually orthogonal. 

(2). The maximal compact subgroup of the group corresponding to s is T n . 
Hence, T = T" D ZJ. Systems C n and BC n have the same Weyl group W = BC n . 
Therefore, H = WT = S n T n . □ 

Let D be a bounded symmetric domain in a complex linear space d (may be, 
reducible) and C 5 be the complex linear span of a maximal abelian subspace in 
c) (thus, we identify with the corresponding space in the Cartan decomposition 
(|5.ip . which is induced by the Cartan involutions in irreducible components). Let 
Autoo(o, D) denote the subgroup of all linear transformations in Aut(£>) which keep 
and each irreducible component of D. 

Corollary 5.3. Let F be a subgroup of S n , G = FT 1 C GL(n,C). Then G 
satisfies condition (2) of Theorem \4-10\ if and only if (V, G) is isomorphic to a pair 
(o, Autoo(o, D)) for a bounded symmetric domain D. 

Proof. All pairs (C n ,S n T n ) appear as (t>, Autoo(t>, -D)) for matrix balls D. It re- 
mains to combine Theorem 14.101 and Proposition [ST2J □ 

It is possible now to describe hulls of iC-orbits in D (with respect to the complex 
structure j) it terms of Proposition 13. 21 The key point is that K is polar in 5: each 
if-orbit meets a orthogonally (i.e., a is a Cartan subspace). This is true, since all 
maximal abelian subspaces are conjugate in by K, ad(a) is symmetric if a £ 5 
and, for a generic a £ a, kerad(a) = a; hence, 

(5.12) [a,g] = a x . 

We may include the linear base in into a base in O as the first n vectors of the 
latter. Then z±, . . . , z n are coordinates in and linear functions in 0. The functions 
[ik in (|0.6[) admit a if- invariant extension to 0: 

(5.13) n k (z) = sup{|(gz)i . . . (gz) k \ : g e K}, 

where k — 1, . . . , n. The following lemma shows that (|5.13p is an extension indeed. 
Lemma 5.4. For z E 0. (|0. 6|) and (|5.13[) coincide. 

Proof. It follows from (15.12|) that any critical point of the linear function Rc z\ on 
the orbit Kz belongs to a. If the lemma is not true, then there exist z € and 
k G {1, . . . , n} such that |(gz)fc| > \zk\- Transformations in S n and T reduce the 
problem to the case z\ > ■ ■ ■ > z n > and k = 1, but then the assumption implies 
that Re z\ attains its maximal value on Kz outside of a. □ 

Proposition 5.5. For any v € 5, Kv — {z 6 f : < f-kiv), k = 1, . . . , n}. 

Proof. Due to (|5.13[) . each ^ is a supremum of absolute values of holomorphic 
polynomials. Hence, the right-hand side is polynomially convex. Thus, it includes 
Kv. The inverse inclusion holds, since each if -orbit intersects by an ii-orbit 
and hulls of ii-orbits are distinguished in by the same inequalities according to 
Proposition 13.21 and Lemma l5~4l □ 
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The functions fj,k can be written in more invariant terms. To do it, note that the 
Weyl group of A a has the form U-[S n in the base OE by (|5.9|) : thus, z k = ctk(z), 
k = 1, . . . ,n, where ak G OE and z G a. Therefore, Zk are eigenvalues of ad(z) 
in the subspace generated by the corresponding root vectors. The problem is to 
distinguish this subspace (in fact, we use a slightly different version). After that, 
functions [ik can be defined as norms of some operators according to the following 
lemma (this observation was used in [12] in another context). 

Lemma 5.6. Let V be a Euclidean space and A be a symmetric nonnegative oper- 
ator in V with eigenvalues Ai > A2 > ■ • • > A m > 0, where m = dim V. Let A Ak be 
its natural extension to the k-th exterior power 

yAk = /yfe y Then 

\\A Ak \\ V Ak = Ai . . . Afe, 

where \\ \\k is the operator norm with respect to the inner product in V Ak . 

Proof. The norm of a nonnegative symmetric operator is equal to its maximal 
eigenvalue. □ 

Let v G Q be semisimple and ir(v) denote the projection onto kerad(u) along 
other eigenspaces of &d(v) (note that ir(v) is a function of ad(t>), since it is the 
residue at zero of the resolvent of ad(v)). Set 

a(v) = ad([v, [v, k]])Tr(v) ad(k), 
Pk{v) = ||a(w) Afe |j B A fe , k = l,...,n. 

The space D is a(u)-invariant and kera(u) 2 6- We assume that g is equipped 
with some A"- invariant inner product, which extends the inner product in 0. It 
follows from the calculation below that a(v) is symmetric and has range ad(k)o. 
Let n = dim a be the rank of the symmetric space D. It is equal to the codimcnsion 
of a generic A-orbit in d. 

Theorem 5.7. For any v £ d, 

Kv — {z G : pk(z) < Pk{v), k=l,...,n}. 

Proof. It is sufficient to prove the assertion for a generic v G 0. Clearly, pk are 
A-invariant. Hence, we may assume v G a. Then, by (|5.9|) and (|5.10p . 

v = v a (e a +f a ). 
where v a G R. According to (|5.2p and (|5.4p . [k, v] — J2 aeE iv a (e a — f«). Thus, 

[«,[«. k]] =Y1 , F ^ hQ 

due to (|5.7|) . Also, (|5.7p implies that ad(zh Q ) keeps and has eigenvalues 0,±2i 
in it for each a G E. Therefore, is ad([u, [v, k]])-invariant and its eigenvalues are 
±4u^,i, a G E. Since ad(k)g = and ir(v)d = a for a generic v G a, the space is 
a(i>)-invariant; moreover, a(v)g = ad(k)a C 0. Thus, a(v) has eigenvalues 0, ±4w^ 
in g. According to Lemma T5. 61 and (|0.6p . 

(5.14) p k {v) = A k ^ 2 k {v) 

for v G and fc = 1, . . . , n. Since and /i^ are A-invariant, (|5.14p holds for all 
v G 5. The theorem follows from Proposition [^3] □ 
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Corollary 5.8. Functions pk, k = 1, . . . , n, are plurisubharmonic in 5 with respect 
to the complex structure j = ad(k). 

Proof. By (|5TT3|l and l[5TTg]) . 

Pk (z) =A k svLv{\{gz)\...{gz)l\ : g € K}. 

The right-hand side is plurisubharmonic, since the functions z\ are j-holomorphic 
and j is if- invariant. □ 

One can get the same functions pk by replacing g with t>, endowed with the 
complex structure j, and a(v) with ad([v,jv])(n(v) + w(jv)). 

Acknowledgement. I thank Anton Pankratiev for helpful comments. 
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